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Augmented Reality Traffic Control Center 



Background of the Invention 



Field of the Invention 



[0001] 



The present invention relates generally to traffic control systems, and 



more particularly to air traffic control systems. 



Related Art 



[0002] 



Operations in conventional traffic control centers, such as, e.g., primary 



flight control on an aircraft carrier, airport control towers, and rail yard control 
towers, are severely impacted by reduced visibility conditions due to fog, rain and 
darkness, for example. Traffic control systems have been designed to provide 
informational support to traffic controllers. 



from detectors and the objects being tracked to show the controller where the objects 
are in two dimensional (2D) space. For example, an air traffic control center in a 
commercial airport, or on a naval aircraft carrier at sea, typically uses a combination 
of radar centered at the control center and aircraft information from the airplanes to 
show the controller on a 2D display, in a polar representation, where the aircraft are 
in the sky. Unfortunately, unlike automobile traffic control systems which deal with 
two dimensional road systems, air traffic adds a third dimension of altitude. 
Unfortunately, conventional display systems are two dimensional and the controller 
must mentally extrapolate, e.g., a 2D radar image into a three dimensional (3D) 



[0003] 



Conventional traffic control systems make use of various information 
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representation and also project the flight path in time in order to prevent collisions 
between the aircraft. These radar-based systems are inefficient, however, at 
collecting and conveying three or more dimensional data to the controller. 

[0004] Conventional systems offer means to communicate with the individual 

aircraft, usually by selecting a specific communication channel to talk to a pilot in a 
specific airplane. This method usually requires a controller to set channels up ahead 
of time, for example, on an aircraft carrier. If an unknown or unanticipated aircraft 
enters the control space, the control center may not be able to communicate with it. 

[0005] What is needed then is an improved system of traffic control that 

overcomes shortcomings of conventional solutions. 

Summary of the Invention 

[0006] An exemplary embodiment of the present invention provides a traffic 

controller, such as an air traffic controller, with more data than a conventional radar- 
based air traffic control system, especially in conditions with low visibility such as 
low cloud cover or nightfall. The system can provide non-visual data, such as, e.g., 
but not limited to, infrared and ultraviolet data, about traffic control objects, and can 
display that information in real-time on displays that simulate conventional glass- 
window control tower views. In addition, the system can track the movements of the 
controller and receive the movements as selection inputs to the system. 

[0007] In an exemplary embodiment, the present invention can be an augmented 

reality system, that may include a display; a sensor for collecting non-visual data 
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associated with traffic control objects in a traffic control space; a computer receiving 
the data from the sensor, and operative to display the data on the display in real time; 
and means for detecting a physical gesture of a traffic controller selecting an traffic 
control object displayed on the display. 

[0008] In an another exemplary embodiment, the present invention can be a 

method of augmented reality traffic control including collecting non-visual data 
associated with traffic control objects in a traffic control space; displaying the non- 
visual data in real time; and detecting a physical gesture of a traffic controller 
selecting one of the traffic control objects displayed. 

[0009] Further features and advantages of the invention, as well as the structure 

and operation of various embodiments of the invention, are described in detail below 
with reference to the accompanying drawings. 



Definitions 

[00010] Components/terminology used herein for one or more embodiments of the 

invention are described below: 

[0001 1] In some embodiments, "computer" may refer to any apparatus that is 

capable of accepting a structured input, processing the structured input according to 
prescribed rules, and producing results of the processing as output. Examples of a 
computer may include: a computer; a general purpose computer; a supercomputer; a 
mainframe; a super mini-computer; a mini-computer; a workstation; a 
microcomputer; a server; an interactive television; a hybrid combination of a 
computer and an interactive television; and application-specific hardware to emulate a 
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computer and/or software. A computer may have a single processor or multiple 
processors, which may operate in parallel and/or not in parallel. A computer may 
also refer to two or more computers connected together via a network for transmitting 
or receiving information between the computers. An example of such a computer 
may include a distributed computer system for processing information via computers 
linked by a network. 

[00012] In some embodiments, a "machine-accessible medium" may refer to any 

storage device used for storing data accessible by a computer. Examples of a 
machine-accessible medium may include: a magnetic hard disk; a floppy disk; an 
optical disk, such as a CD-ROM or a DVD; a magnetic tape; a memory chip; and a 
carrier wave used to carry machine-accessible electronic data, such as those used in 
transmitting and receiving e-mail or in accessing a network. 

[0001 3] In some embodiments, "software" may refer to prescribed rules to operate 

a computer. Examples of software may include: code segments; instructions; 
computer programs; and programmed logic. 

[00014] In some embodiments, a "computer system" may refer to a system having 

a computer, where the computer may comprise a computer-readable medium 
embodying software to operate the computer. 

Brief Description of the Drawings 
[0001 5] The foregoing and other features and advantages of the invention will be 

apparent from the following, more particular description of exemplary embodiments 
of the invention, as illustrated in the accompanying drawings wherein like reference 
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numbers generally indicate identical, functionally similar, and/or structurally similar 

elements. The left most digits in the corresponding reference number indicate the 

drawing in which an element first appears. 
[00016] FIG. 1 depicts an exemplary embodiment of an augmented reality air 

traffic control system according to the present invention; 
[0001 7] FIG. 2 depicts a flow chart of an exemplary embodiment of a method of 

augmented reality traffic control according to the present invention; and 
[00018] FIG. 3 depicts a conceptual block diagram of a computer system that may 

be used to implement an embodiment of the invention. 



Detailed Description of Exemplary Embodiments of the Present Invention 
[00019] A preferred embodiment of the invention is discussed in detail below. 

While specific exemplary embodiments are discussed, it should be understood that 
this is done for illustration purposes only. A person skilled in the relevant art will 
recognize that other components and configurations can be used without parting from 
the spirit and scope of the invention. 
[00020] As seen in FIG. 1 , in an exemplary embodiment, an air traffic control 

system 100 can use different types of sensors and detection equipment to overcome 
visibility issues. For example, the system 100 can use infrared (IR) cameras 102, 
electro-optical (EO) cameras 104, and digital radar 106, alone or in combination, to 
collect visual and non-visual data about an air traffic control object, such as, e.g., but 
not limited to, airplane 101. Additional sensors can include, e.g., but are not limited 
to, a radio-frequency image sensor, RADAR, LIDAR, a millimeter wave imaging 
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sensor, an acoustic sensor, a digital infrared camera, a digital ultraviolet camera, and 
high-resolution radar. The sensor data may be provided to the virtual reality (VR) or 
augmented reality system 108, which may process with computer 1 18 the sensor data, 
and may display the data 1 10 in visual form to the controller 1 12, even when 
visibility is limited. In an exemplary embodiment, the data 110 can be presented to 
the controller 1 12 in an immersive virtual reality (VR) or augmented reality system 
108 using large flat panel displays 1 14a-e (collectively 1 14) in place of, or in addition 
to, glass windows, to display the data 1 10 in a visual format. Then, regardless of the 
external conditions, the controller 1 12 can see the flight environment as though the 
weather and viewing conditions were bright and clear. In another exemplary 
embodiment, the data 1 10 can be displayed to the controller 1 12 in a VR helmet worn 
by the controller 1 12, or other display device. 
[00021] An exemplary embodiment of the present invention can also make use of 

augmented reality (AR) computer graphics to display additional information about the 
controlled objects. For example, flight path trajectory lines based on an airplane's 
current speed and direction can be computed and projected visually. The aircraft (or 
other control objects) themselves can be displayed as realistic airplane images, or can 
be represented by different icons. Flight information, such as, e.g., but not limited to, 
flight number, speed, course, and altitude can be displayed as text associated with an 
aircraft image or icon. Each controller 1 12 can decide which information he or she 
wants to see associated with an object. The AR computer system 108 can also allow 
a controller 1 12 to zoom in on a volume in space. This is useful, for example, when 
several aircraft appear "stacked" too close together on the screen to distinguish 
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between the aircraft. By zooming in, the controller 1 12 can then distinguish among 
the aircraft. 

[00022] An exemplary embodiment of the present invention can also provide for 

controller input such as, e.g., but not limited to, access to enhanced communication 
abilities. A controller 112 can use a gesture detection device 1 16 to point, for 
example, with his or her finger, to the aircraft or control object with which he or she 
wants to communicate, and communication may be opened with the aircraft by the 
system. The pointing and detection system 116 can make use of a number of 
different known technologies. For example, the controller 1 12 can use a laser pointer 
or a gyro-mouse to indicate which aircraft to select. Alternatively, cameras can 
observe the hand gestures of the controller 1 12 and feed video of a gesture to a 
computer system that may convert a pointing gesture into a communication opening 
command or other command. The controller 1 12 can alternatively wear a data glove 
that can track hand movements and may determine to which aircraft the controller is 
pointing. Alternatively, the gesture detection device 1 16 may be a touch-sensitive 
screen. 

[00023] In addition to the various exemplary sensors 1 02- 1 06 that may be used as 

inputs to the system 108, the various exemplary sensors 102-106 track objects of 
interest in the space being controlled. Information from other sources (such as, e.g., 
but not limited to, flight plans, IFF interrogation data, etc.) can be fused with the 
tracking information obtained by the sensors 102-106. Selected elements of the 
resulting fused data can be made available to the controllers 1 12 through both 
conventional displays and through an AR or VR display 1 10, 1 14 which may 
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surround the controller 1 12. The location and visual focus of the controller 1 12 can 
be tracked and used by the system 108 in generating the displays 1 10, 1 14. The 
physical gestures and voice commands of controller 112 can also be monitored and 
may be used to control the system 108, and/or to link to, e.g., but not limited to, an 
external communications system. 
[00024] In an exemplary embodiment, the detected physical gesture of the 

controller 1 12 may be used to open a computer data file containing data about the 
selected air traffic control object. The computer data file may be stored on, or be 
accessible to, computer 118. The data in the computer data file may include, for 
example, a passenger list, a cargo list, or one or more physical characteristics of the 
selected air traffic control object. The physical characteristics may include, but are 
not limited to, for example, the aircraft weight, fuel load, or aircraft model number. 
The data from the computer data file may then be displayed as a textual annotation on 
the display 114. 

[00025] In an exemplary embodiment, the present invention can be used, for 

example, for augmenting a conventional aircraft carrier Primary Flight (PriFly) 
control center. A PriFly center can use head-mounted display technology to display 
track annotations such as, e.g., but not limited to, flight number, aircraft type, call 
sign, and fuel status, etc., as, e.g., a text block projected onto a head mounted display 
along a line of sight from a controller 1 12 to an object of interest, such as, e.g., but 
not limited to, an aircraft. For example, the head mounted display can place the 
information so that it appears, e.g., beside the actual aircraft as the aircraft is viewed 
through windows in daylight. At night or in bad weather, the same head mounted 
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display can also be used to display, e.g., real-time images obtained by exemplary 
sensors 102-106, such as, e.g., but not limited to, an infrared camera 102 or low light 
level TV camera imagery at night, to provide the controller 1 12 with the same visual 
cues as are available during daylight. 

[00026] In an exemplary embodiment, a position, visual focus, and hand gestures 

of the controller 1 12 can be monitored by, e.g., a video camera and associated 
processing system, while voice input might be monitored through, e.g., a headset with 
a boom microphone. In addition to visual focus, voice commands, and hand gestures 
being used to control the augmented reality control tower information processing 
system 100, a controller 1 12 can point or stare at a particular aircraft (which might be 
actually visible through the window or projected on the display) and may order the 
information processing system 108 via gesture detection device 1 16 to, e.g., open a 
radio connection to that aircraft. Then the controller 1 12 could, e.g., talk directly to 
the pilot of the aircraft in question. When the controller 1 12 is finished talking with 
that pilot, another voice command or a keyboard command, or other input gesture 
could close the connection. Alternatively, for aircraft with suitable equipment, the 
controller 112 can dictate a message and then tell the information processing system 
to transmit that message to a particular aircraft or group of aircraft. Messages coming 
back from such an aircraft could be displayed, e.g., beside the aircraft as a text 
annotation, or appear in a designated display window. 

[00027] An exemplary embodiment can use an immersive virtual reality (VR) 

system 108 to present and display sensor 102-106 imagery and computer 
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augmentations such as, e.g., text annotations. Such a system can completely replace a 
conventional control center along with its windows. 
[00028] An exemplary embodiment of the present invention can also be used to 

control, e.g., train traffic at train switching yards and crossings. Similarly, the 
immersive VR system 108 may be used in other traffic control management 
applications. 

[00029] Some exemplary embodiments of the invention, as discussed above, may 

be embodied in the form of software instructions on a machine-accessible medium. 
Such an exemplary embodiment is illustrated in FIG. 3. The computer system 1 18 of 
FIG. 3 may include, e.g., but not limited to, at least one processor 304, with 
associated system memory 302, which may store, for example, operating system 
software and the like. The system may further include additional memory 306, which 
may, for example, include software instructions to perform various applications and 
may be placed on, e.g., a removable storage media such as, e.g., a CD-ROM. System 
memory 302 and additional memory 306 may be implemented as separate memory 
devices, they may be integrated into a single memory device, or they may be 
implemented as some combination of separate and integrated memory devices. The 
system may also include, e.g., one or more input/output (I/O) devices 308, for 
example (but not limited to), keyboard, mouse, trackball, printer, display, network 
connection, etc. The present invention may be embodied as software instructions that 
may be stored in system memory 302 or in additional memory 306. Such software 
instructions may also be stored in removable media (for example (but not limited to), 
compact disks, floppy disks, etc.), which may be read through other memory 306, or 
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an I/O device 308 (for example, but not limited to, a floppy disk drive). Furthermore, 
the software instructions may also be transmitted to the computer system via an I/O 
device 308, including, for example, a network connection; in this case, the signal 
containing the software instructions may be considered to be a machine-accessible 
medium. 

[00030] While various embodiments of the present invention have been described 

above, it should be understood that they have been presented by way of example only, 
and not limitation. Thus, the breadth and scope of the present invention should not be 
limited by any of the above-described exemplary embodiments, but should instead be 
defined only in accordance with the following claims and their equivalents. 
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